opcua: rescan while disconnected, heal PLC_COMMS_LOST on connect, and name a plugin freeze-frame's source - #660

A freeze-frame captured for a plugin-backed entity (#564) reaches a client
with an empty topic and an empty message_type. That is correct, the values
are the plugin's live entity data and not a ROS message, but it leaves the
snapshot with no field at all naming its origin. Two of these frames from
different bridges are indistinguishable, and a frame is indistinguishable
from a topic capture whose metadata went missing.

Frame now carries the capture path that read it, and it is served as
x-medkit.source: plugin_data_provider for a read through the owning
plugin's DataProvider, plugin_x_plc_data_route for the in-process dispatch
of the plugin's own x-plc-data route (bridges that export no
DataProvider). topic and message_type are left empty rather than
overloaded, since neither names a ROS topic here.

The field is omitted, not emptied, when the capture named no path, so a
fault-manager freeze-frame taken from a real topic is unaffected and
carries its topic and message_type as before.

The gateway runs four nodes inside its own process, all named after
itself: the gateway node, "<node>_sub" for the subscription executor,
"<node>_fault_clients" for the fault-service transport, and
"<node>_lifecycle_state_reader" for the lifecycle reader. None of them
starts with an underscore, so the ROS 2 hidden-node convention does not
cover them, and runtime introspection returned all four as ordinary Apps.
The gateway advertised its own plumbing as diagnosable entities, and an
operator browsing /api/v1/apps saw four entries that answer nothing
useful.

There were two half-answers to the same question. count_peer_nodes knew
the gateway's own FQN plus "_sub" and "_fault_clients" but not the
lifecycle reader, and the app filter knew only the underscore rule, so it
dropped none of the four. Both now go through one predicate,
is_own_gateway_node, so a fifth helper is declared in one place instead
of two.

The match is exact per suffix, never a prefix test: a genuine peer named
"<node>_monitor" or "<node>2" must stay visible, and hiding a real node is
the worse error. A fault_manager sharing the process is not ours either
and stays listed. Remote entities are left alone: a peer's helper nodes
carry the same fully qualified names and are the peer's own filter's
business.

Every existing opcua docker scenario pins OPCUA_ENDPOINT_URL, which
short-circuits discovery, so none of them can reach the failure this
covers: the gateway and the PLC power on together, the start-up scan runs
while the PLC is still booting, and the plugin is left retrying its
fallback endpoint.

The scenario starts the gateway first with discovery on and no endpoint
configured, asserts it settled on the fallback endpoint with no session,
then brings an OPC-UA server up on the same subnet and asserts the
endpoint is adopted within two re-scan intervals. It also asserts the
container never restarted, since a restart would satisfy the endpoint
check while proving nothing: a restart is exactly what used to be needed.

The network is created with an explicit /24 so the read-only sweep stays
254 hosts and finishes in seconds.

…cted

A gateway that starts before its PLC runs the discovery rescan for the
life of the outage, and several things it does in that state were wrong.

Component identity. With no node map the SOVD component is named from
the device. When the start-up connect fails there is no device to ask,
so the name comes from the fallback endpoint and an empty DeviceInfo,
and it was then pinned forever: after discovery adopted the real PLC the
component still served opcua-<fallback host> while introspect() reported
the adopted endpoint. The identity is now re-derived on the first poll
of a new session (config-less mode only, an explicit node map still owns
the name), the derived alarms entity follows the rename, and the change
is logged at INFO. The docker race scenario gained a config-less pass
that asserts the rename against a real server.

Rescan cadence. The cadence was stamped when a sweep STARTED, so a sweep
of a legal /16 (minutes at the defaults) made the next one due the
moment it returned: the poll thread swept back to back and the reconnect
attempt dropped to one per sweep. It is now stamped when the sweep ends.
NetworkDiscovery::run() also takes a cancel predicate, bound to the
shutdown flag and checked before each probe and between the sweep and
identify phases, so shutdown() no longer has to wait out a sweep.

Reconnect backoff. The rescan is consulted once per reconnect attempt
and attempts are spaced by the exponential backoff, so the real cadence
was max(interval_s, backoff) while the README, the header and the
start-up log all said "every interval_s". The backoff ceiling is now
capped at the rescan cadence while discovery is rescanning.

interval_s. An unset interval and an explicit 0 both mapped to the 30 s
default, so there was no way to keep discovery on and stop rescanning.
They are now distinct: unset takes the default, an explicit 0 leaves the
start-up scan one-shot, and a negative value is refused with a warning
that no longer claims the kept default is one-shot. The start-up line no
longer promises a loop that rescans "every 0s" either.

Discovery report. A pass re-emitted its whole report every rescan, so a
site with a secured-only server logged the same WARN every 30 s for the
life of the process. A pass whose outcome matches the previous one now
reports at DEBUG. The first pass, and every changed outcome, still
reports at INFO/WARN.

Connect-time clear. It is a link-state clear, not an operator resolving
a root cause, so it now sets skip_correlation_auto_clear and cannot
cascade-clear the symptom faults a rule attributes to PLC_COMMS_LOST.
The poller's own clear on a successful reconnect is the same event and
does the same. Those clears were also buffered unconditionally while the
fault manager was unmatched, so a flapping link pushed real alarm
reports out of the bounded buffer: the buffer now keeps at most one
pending clear per fault code, evicts a clear before a report, and
refuses a clear rather than dropping a report.

The two Frame::source constants had no assertion from a capture: only a
hand-built frame in the merge helper's test named one, so swapping the
DataProvider and route values left the whole suite green. The route and
DataProvider loss-of-comms tests now each assert the constant their own
path must produce, plus the literal wire value - symbol against symbol
stays equal when the two constants are swapped, and that string is what
every x-medkit.source consumer reads. The merge test that omits the key
keeps its place as a helper contract for a frame a caller built without
naming a path - the capture paths always name one - and says so instead
of standing in as a control for them.

The peer-node count test listed only two of the three helper nodes the
gateway creates in its own process, so a lone gateway with a lifecycle
reader would have counted a peer and skipped the empty-graph warning.

Document the entity-frame source field in the REST fault snapshot
reference: an entity frame carries no topic or message type, so source
is the only provenance a consumer gets.

…y what is re-derivable

The per-entity SOVD route DELETE /{entity}/faults/{code} lands on
FaultProvider::clear_fault for a plugin-owned entity, which is the branch
the gateway takes instead of its own. The gateway sets
skip_correlation_auto_clear there so an operator scoped to one entity
cannot cascade-clear correlated symptoms reported by apps in other
entities, and the ClearFault contract documents that guarantee, but this
plugin sent the flag off and reopened the hole wherever a PLC is
involved. It now sets the flag, and the reason each call site sets or
clears it travels with the call: a ClearOrigin says whether the device
reported the condition inactive (a real resolution, cascade kept), the
link came back, or an operator cleared through the scoped route.

The pending-dispatch buffer used that same distinction too bluntly. It
gave up any clear before any report, but only the link-state clear is
re-derivable - the next reconnect sends it again. A device alarm's
inactive edge is as one-shot as its raise, so evicting it left the flush
replaying the raise with nothing behind it and the fault standing while
the device said inactive. Ranking is now by re-derivability: the
link-state clear is what a full buffer gives up first, everything else
ages out oldest-first as it did before.

The start-up discovery sweep claimed to be cancellable through the
shutdown flag, but nothing can set that flag while it runs: it happens
inside set_context(), during node construction, before the executor the
gateway shuts down from ever spins. A SIGTERM during a wide sweep
therefore waited the sweep out. Both sweeps now ask one predicate that
also reads rclcpp::ok(), which rclcpp's own signal handler turns false,
so the start-up sweep ends on the signal and the rescan keeps ending on
shutdown() as well.

Also: a rescan sweep that throws now stamps the cadence on its way out,
or the next poll iteration would immediately start another one; the
start-up log line no longer states the wrong reason for reporting in
full; and the package changelog records this branch.

Measuring that start-up sweep also showed the gateway logging nothing at
all while it ran: the discovery report is buffered so a repeated rescan
can be reported at DEBUG, and the "scanning [subnets]" announcement had
been swept up with it. A minutes-long sweep with no output reads as a
hung process, so the announcement is sent before the sweep again, with
the same first-pass INFO / rescan DEBUG levelling on its own.

…e wire

Four call sites decide whether a ClearFault may cascade, and only the
scoped SOVD DELETE was pinned: swapping the origin at any of the other
three left the suite green. The two that need a live session are now
driven against the test_alarm_server fixture with a real fault manager on
the other end, so the flag is read off the wire. A successful connect
must clear PLC_COMMS_LOST without cascading, and the same test drives the
fixture's own CLI to fire and clear a condition, because a clear the
device itself reported is a resolution at the source and must keep the
cascade. Having both cases in one test is what makes each flag a decision
rather than a constant. The fixture harness gained a stdin pipe to send
those commands, the way the docker scenario already drives it through a
FIFO.

The poller's own clear travels the same callback as every device alarm,
so the rule that tells them apart moved into clear_origin_for_signal and
is tested on both branches, exact code match included.

The sweep's cancel predicate had the same shape of hole: it is private
and no test reached it, so reverting it to the shutdown flag alone left
everything green while a SIGTERM during the start-up sweep would again
have to wait the sweep out. The rule is now the static
discovery_cancelled_for, tested on both inputs, with the member reduced
to reading the two values off the process. A second test shows the input
is real by shutting a private rclcpp context down and reading
rclcpp::ok() back. The comment on the cancellation test no longer claims
it exercises the plugin's own predicate, which it never did.

Also: the remaining prose semicolons on this branch (two comments and
five operator-visible log strings) are periods and commas now, and the
Refused outcome's doc comment says what it means.

The app filter removes only the helper nodes the gateway runs inside its
own process - "<gateway>_sub", "<gateway>_fault_clients" and
"<gateway>_lifecycle_state_reader". The gateway's own node stays an App:
its ROS parameters are what SOVD serves as that App's configurations, and
two gateways watching one graph have to agree on what is on it.

A feature test launches a gateway with its real process node names - the
suite's usual launch remaps all four to one string - and pins both halves:
the gateway is listed, addressable and configurable, and each helper is on
the graph but absent from the app list.

An accepted stop leaves the goal CANCELING or CANCELED, and which of the
two a caller observes depends on how fast the server winds down. The
body renders the status the handler read, which is at or before the one
the test reads afterwards, so a goal still CANCELING pins the body
exactly while a goal already CANCELED admits either rendering. What stays
ruled out is a goal that is still running or completed anyway.

The e2e tests that drive the plugin against a live fault-manager stub spin
a MultiThreadedExecutor on a thread of their own. A gtest ASSERT_* returns
from the middle of the test body, and a std::thread destroyed while still
joinable calls std::terminate: the run ends in SIGABRT, the assertion
message that names the real failure never reaches the report, and the
alarm-server child outlives the test holding its port.

An RAII guard owns the thread and runs cancel -> join, so the assertion
failure is what the report shows. Its stop() is idempotent, so the tests
still end the spin where they want the executor quiet.

The fixture's open62541 is built with UA_MULTITHREADING 0, so the server
may only be used from one thread. The command handler ran on a second
thread and wrote condition fields and triggered condition events there
while UA_Server_run iterated on the main thread; the mutex it held was
taken nowhere else, so it synchronised the handler against itself and
nothing against the server.

The reader thread now only queues command lines. The server is driven by
UA_Server_run_startup / run_iterate / run_shutdown, and each queued
command runs between iterations on that thread, so the address space and
the Alarms & Conditions subsystem have a single user. The reader ends on
EOF, which the harness produces by closing its write end of the pipe
before it signals the server.

The three helper FQNs are not all built the same way. The subscription
executor passes the gateway's own namespace, so its node follows the
gateway; the fault-client and lifecycle-reader nodes are built from the
gateway's node name alone and take the process default namespace. A remap
naming the gateway alone - "-r <gateway>:__ns:=/x" - moves the gateway and
the subscription node and leaves the other two behind, and the gateway
then served both of them as diagnosable apps and counted them as peers.

The predicate derives each helper's FQN from its own creation site and
matches both spellings for the two that do not follow the namespace.

…ared

The undeclared-node warning on GET /health is an instruction - "Declare
them in the manifest" - and it was naming the three nodes the gateway runs
in its own process. Declaring one produces a manifest app that the app
filter then removes, so the two halves contradicted each other.

The orphan scan skips the helper nodes, using the same predicate the app
filter uses, and a declared app bound to one of them is dropped with a
warning that names it instead of a debug line. A feature test drives a
hybrid gateway with "unmanifested_nodes: error" and pins both: the report
names an undeclared node of the test's own and none of the helpers.

The gateway creates its helper nodes at three points of start-up, two of
them after the first refresh and after /health starts answering. A settle
gate that waits for the gateway's own node to appear can therefore read a
list built from a graph that held no helper at all, and the absence
assertions then hold whatever the filter does.

The settle sequence now proves its own order: all three helpers on the
graph first, then a witness node created after them, then an /apps
snapshot that lists the witness - such a snapshot was built from a graph
that held the helpers too. The three stage budgets also sum well inside
the file's ctest timeout, so a failure reports its assertion instead of
being killed unnamed.

The config reference documents the rule: which nodes are filtered, why the
gateway's own node stays an App, how the namespaces work out, and what the
switch re-exposes when turned off.

…ase that failed

The quit branch closed this process's own stdin descriptor and claimed
that was what released the reader. It is not: close(2) does not unblock a
reader already parked in read(2), which a timed run confirms - with the
writer holding the pipe for eight seconds after quit, the process ended
after eight seconds, not at once. The call is gone and the comment names
the real terminator, the EOF the writer produces by closing its end.

The exit diagnostic named UA_Server_run, which the loop no longer calls.
Startup and shutdown each report their own bad status now. run_iterate has
no status to report - its return is how long the server may idle - so it
bounds the pause between iterations instead of being dropped.

Both cancel() and join() can throw and a destructor is implicitly
noexcept, so an escape from the guard's destructor would be the
std::terminate the guard exists to prevent. The destructor contains them
and prints what happened; by then the test has already recorded its
verdict, which is what the run has to report.

stop() also sets its idempotence flag after the join rather than before,
so a throwing cancel leaves the object willing to try again instead of
holding a thread nobody will join.

The fixture server succeeds at the requested sequence length, one element
per 100 ms tick, so the order sets how long the goal stays cancellable. At
20 it finishes on its own in under two seconds, which a loaded or
instrumented runner can spend on the create plus the cancel round trip,
and the assertions then describe a goal that completed rather than one
that was stopped. 50 is the largest order the server accepts.

The tracked-goal helper returns its optional instead of dereferencing it
after a non-fatal expectation, and the caller asserts, so a missing goal
is a readable failure rather than undefined behaviour.

The condition is a static misconfiguration - an app declared against one of
the gateway's helper nodes stays declared - while refresh_cache() runs on
every graph event and again on the backstop cadence, which the integration
fixtures set to one second. The warning repeated for the life of the
process. It is now gated on the set of offending apps changing, the same
one-shot discipline the entity-cache warning in the same function uses.

The predicate's doc comment and the config reference also write down what
the root-namespace spelling costs: a helper-named node at the root is
plumbing whichever gateway created it, because two gateways that keep the
default node name and differ only in namespace build the same fully
qualified name for it.

…ateway

The three helper node names take two shapes, and only one of them was
covered end to end. A remap naming the gateway alone moves it and the
subscription node into a namespace and leaves the fault-client and
lifecycle-reader nodes at the root, which is the shape the predicate
reconstructs rather than reads.

The new feature test launches exactly that invocation, asserts the split is
really on the graph, and pins the app list to the gateway plus the test's
own two nodes.

…es early

Two ways the guard could still reach std::terminate through its thread
member, which no catch in its destructor can intercept. A cancel() that
throws returned from stop() before the join, leaving the thread joinable.
And a cancel issued before spin() has begun is refused and lost, so the
thread spins on and the join never returns - a test that asserts early
enough hangs until ctest kills it.

The cancel is now contained and re-issued until the spin function itself
reports it returned, and only then is the thread joined. A test injects a
cancel that throws and reaches the end of the body, which under the old
shape ended the binary with SIGABRT instead.

The new predicate's declaration sat between filter_internal_node_apps' doc
block and the function it documents, so doxygen attached that block to the
predicate and reported four parameters it does not take.

…ep the PLC's order

The PLC_COMMS_LOST clear is a decision the plugin takes on the poll thread,
only while the link is up: it asks the fault manager who reported the
standing row and clears it only when every source is an id this bridge
reports under (the current component id and the stand-ins it assigned).
A row two bridges hold stays for an operator. One fault-store read is
outstanding at a time, stamped with the probe and the connection
generation; a read the store does not answer within
fault_service_timeout_ms is dropped and taken again; an answer from before
a link drop is discarded and the decision owed again; a link-state clear
the bounded buffer could not keep is owed and decided once the sink is
ready.

The pending buffer coalesces a clear onto an earlier pending clear for the
same code only when nothing was reported for that code in between, so
report, clear, report, clear flushes in that order.

Identity in config-less mode: the connect hook re-derives the component
name from the device before the session subscribes, so replayed
conditions land on the new entity; a nameplate replaces a stand-in and
nothing replaces a nameplate; an empty read renames a stand-in only when
the endpoint changed; reads are budgeted per session; conditions the
poller already pinned move to the new entity; the event path reads a copy
of the alarm routing owned by the poller.

The binding: the plugin remembers the ApplicationUri of the server a
session reached. A rescan while disconnected looks for that server at any
address and refuses every other; a different server at the bound address
is dropped at connect. The first adoption, a configured endpoint_url and
a server without an ApplicationUri are unconstrained, and the README says
so. A rescan that throws is caught on the poll thread. The cancel
predicate reads the node's own context.

…e restart checks

The config-less pass asserts that the stand-in component left /components
in the same sample that shows the renamed one. The containers run with no
restart policy, so a gateway that dies is reported by the endpoint and
component assertions; the RestartCount checks are gone. The deadline
follows RESCAN_INTERVAL_S like its siblings.

…, and own every reconnect

The plugin writes the ApplicationUri it is bound to into discovery.binding_file
(default /var/lib/ros2_medkit/opcua/binding, env OPCUA_DISCOVERY_BINDING_FILE,
empty disables) when the binding is established and reads it in configure(),
so a restart keeps the identity. The start-up sweep and every rescan then
select only the bound server, whatever its address, and a different server at
the bound address is dropped at connect with the outage left standing; the
refusal names the file as the recommissioning gesture. A configured
endpoint_url neither reads nor writes the file, and a server that publishes no
ApplicationUri is never persisted. The file is written through a temporary,
fsynced and renamed over the resolved target, holds one trimmed line, and a
BOM, a control character or a second line is refused with a warning.

The reconnect arm owns every reconnect: after each client iterate the session
state is read, a session the library re-opened underneath is dropped and handed
to the arm, which connects through its own path, runs the binding check and
re-creates the alarm subscription. Every path that marks the client
disconnected drops the dead session's subscription bookkeeping, so the
re-created monitored item delivers events again after a PLC reboot.

Unit tests cover the file's format and the write contract; end-to-end tests
against the fixture cover a restarted plugin refusing a foreign server at the
bound address, a configured endpoint_url ignoring the file, the start-up
sweep selecting the bound server over a lower address, a swap and an outage
under a config-less session, and a device alarm delivered after the bound
server reboots.

The identity E2E's ctest timeout is sized to one stuck test's wait budget on
top of twice the idle pace, with the arithmetic at the registration.

…check each built item

test_a_scoped_item_says_what_its_templated_sibling_says compares every entity
in both listings, counts built items per entity type, collection and scope,
requires a built item for every listed resource, and checks that the item at
each key names that resource. test_every_advertised_collection_is_served
follows every app's hrefs and requires a cache-built href per entity type.
The class waits for calibration's operation through REQUIRED_OPERATIONS and
polls temp_sensor's data with the discovery budget before comparing, so the
counters do not depend on how fast the runner propagates a service.

…identity check covers

The README binding section carries the file's path and format, the
discovery-only rule, the clearing gesture, the start-up sweep and rescan
behaviour, the reconnect ownership and the shapes it was proven in, the
probing cadence at a refused address, the last-eight refused URIs, one file
per plugin instance, the mount needed for a re-created container, and the
configuration and environment rows.

… the plugin's own threads

The ReportFault, ClearFault and GetFault clients live in a callback group
that is not added to an executor together with the node. A single-threaded
executor owned by the plugin serves that group from a thread the plugin
starts in set_context. shutdown stops the poller, stops and joins that
thread, and then drops all three clients on the calling thread, so a client
destructor never runs on a gateway executor thread while another plugin or
the gateway creates an entity on the same node. The GetFault answer is
parked from the plugin's thread under the probe state's mutex, as before.

The client thread also ends once the node's context has shut down.
set_context is documented as called once per plugin instance.

A new end-to-end case builds two plugin instances in sequence on one node
while the gateway's executor spins, and checks that the first shutdown
joins the client thread and that the second instance's clients reach the
fault services.

EntityDetailStatusLinkTest and LifecycleHandlersWithProviderTest initialise
rclcpp when it is not running, and their TearDownTestSuite now shuts it down
the way LifecycleHandlersTest does. The test process exits with no rclcpp
context left initialised, so static destruction does not run alongside live
middleware threads.

…and name the test stubs' parameters

The GetFault answer and the fault_service_timeout_ms config value are read
through const references, and the persisted-binding and rescan results are
returned from non-const locals so they are moved out. The severity band
order check is written as the direct comparison of the integer bands.

The GetFault response callback keeps its future by value, because rclcpp
offers the response-callback overload of async_send_request only to a
callable taking exactly SharedFuture; the check is suppressed at that line
with the reason next to it.

In the identity tests, the stub definitions name every parameter, the child
process fixture and the rclcpp scope guard are neither copyable nor movable,
and the fault store stub's service callbacks take their arguments by const
reference.